Informed Initial Policies for Learning in Dec-POMDPs

Decentralized partially observable Markov decision processes (Dec-POMDPs) offer a formal model for planning in cooperative multi-agent systems where agents operate with noisy sensors and actuators and local information. While many techniques have been developed for solving DecPOMDPs exactly and approximately, they have been primarily centralized and reliant on knowledge of the model parameters. In real world scenarios, the model may not be known a priori, and a centralized computation method may not be feasible or desirable. Simple distributed reinforcement learning, particularly Qlearning (Sutton and Barto 1998), can address both of the above limitations. Q-learning agents can learn mappings from their own action-observation histories (Hi) to their own actions (Ai), via a quality function (Qi : Hi×Ai 7→ R) that evaluates the long-term effects of selecting an action after observing a history. We investigate two approaches to applying Q-learning to Dec-POMDPs: one in which agents learn concurrently (QConc), and one in which agents take turns to learn the best responses to each other’s policies (Q-Alt). In both Q-Conc and Q-Alt, an agent that is learning follows a dynamic policy (i.e. a policy that changes as the agent learns); however, in Q-Alt, each agent that is not currently learning follows a static policy. Thus, in Q-Alt an agent that has not yet learned a policy needs an initial policy to follow. There are simple ways to choose an arbitrary initial policy in constant time, e.g., choosing a pure policy at random, but such methods do not consider the transition, observation, and reward structure of the Dec-POMDP. As the main contribution of this paper, we propose a simple and principled approach to building an initial joint policy that is based upon the transition and reward (but not the observation) functions of a Dec-POMDP. To lay the groundwork, we first discuss how to compute this initial joint policy in a centralized, model-based fashion. We then discuss how agents can learn such policies in a distributed, model-free manner, and then demonstrate for two benchmark problems that Q-Alt initialized with this informed policy (QIP-Alt) produces better joint policies than Q-Conc and Q-Alt initial-